(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 




llll III III II 


minium 


1111 


mum 


hi 


illinium 



(19) World Intellectual Property Organization 

International Bureau 

(43) International Publication Date (10) International Publication Number 

31 January 2002 (31.01.2002) PCT WO 02/08023 A2 

(51) International Patent Classification 7 : B60R 21/01, Mirada, CA 90638 (US). DUN BRIDGE, Barry; 3614 

G08B 21/00 Courtney Way, Torrance, CA 90505 (US). SHREVE, 

Gregory, A.; 1201 South Trotwood Avenue, San Pedro, 

(21) International Application Number: PCT/US0 1/2 1700 CA 90732 < us >* 

(22) International Filing Date: 10 July 2001 (10.07.2001) (74) A g ents: SCHIVLEY, G., Gregory et ah; Harness, Dickey 

& Pierce P.L.C., P.O. Box 828, Bloomrleld Hills, MI 48303 



(25) Filing Language: English 

(26) Publication Language: English 



(US). 

(81) Designated States (national): JP, KR. 



(84) Designated States (regional): European patent (AT, BE, 
(30) Priority Data: CH^ CY , DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, 

09/621,160 21 July 2000 (21.07.2000) US NL, PT, SE, TR). 

(71) Applicant: TRW INC. [US/US]; One Space Park, Re- Published: 

dondo Beach, CA 90278 (US). — without international search report and to be republished 

upon receipt of that report 

(72) Inventors: BERENZ, John, J.; 442 S. Arcadia Drive, 

San Pedro, CA 90731 (US). MCIVER, George, W.; 801 For two-letter codes and other abbreviations, refer to the "Guid- 
N. Paulina Avenue, Redondo Beach, CA 90277 (US). ance Notes on Codes and Abbreviations" appearing at the begin- 
NIESEN, Jospeh, W.; 15840 Shady wood Court, La ning of each regular issue of the PCT Gazette. 



(54) Title: APPLICATION OF HUMAN FACIAL FEATURES RECOGNITION TO AUTOMOBILE SAFETY 




< 

(57) Abstract: An imaging system (50) for providing vehicle safety features that employs face recognition software to identify and 
A track a person. The system (50) employs infrared emitters (30) that emit an infrared signal along a predetermined field-of-view, 
and an infrared sensor (34), such as a CMOS sensor used as a video signal array, that receives reflected infrared illumination from 
objects in the field-of-view. A processor (52) including the face recognition software, is employed to detect human faces to identify 
and track the person. Once a face is detected, it can be compared to a data base to identify the person. Various applications for the 
imaging system (50) for providing vehicle safety features include identifying the driver or passenger for personalizing the vehicle' s 
^ airbags, providing pre-crash collision avoidance, providing blind spot detection, providing vehicle crash recording, and providing a 
warning signal if the driver appears drowsy. 



WO 02/08023 PCT/US01/21700 



APPLICATION OF HUMAN FACIAL FEATURES RECOGNITION 
TO AUTOMOBILE SAFETY 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to an imaging system in a vehicle for 
detecting the presence of a person and, more particularly, to an imaging 
5 system for use in a vehicle that identifies a human face, and uses this 
information for vehicle safety purposes. 

2. Discussion of the Related Art 

The National Highway Traffic Safety Administration requires passenger 
side airbags on all passenger cars manufactured after 1998. Airbags have 

1 0 saved thousands of lives, but safer airbag deployment systems can save even 
more lives. Proposals have been submitted by the National Highway Traffic 
Safety Administration to require advanced airbags that are safer and more 
efficient Thus, there is a need in the art for an advanced airbag deployment 
system that determines airbag deployment depending on the detection, identify 

1 5 and tracking of a person in the passenger seat of the vehicle. 

Future airbag deployment systems will be able to identify and track a 
person in the passenger seat of the vehicle. Based on this information, the 
system will provide signals whether to fire the airbag, fire the airbag at low 
velocity or fire the airbag at normal high velocity during a crash event, 

20 depending on whether a person is seated in the passenger seat, the size of the 
person and the position of the person relative to the airbag deployment door. In 
one example, if a person is detected in the passenger seat, and is some small 
distance (for example, within three inches) from the airbag door, then the airbag 
does not fire during a crash event. If a person is detected in the passenger 

25 seat, and is close (for example, between three and eight inches) to the airbag 
door, the airbag is fired at a lower velocity during a crash even. If the person is 
detected in the passenger seat, and far enough away (for example, more than 
eight inches) from the airbag door, then the airbag is fired at normal high 
velocity during a crash event. 
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Present passenger sensing systems typically include ultrasonic, weight, 
infrared and/or electromagnetic sensors to detect a passenger for airbag 
deployment Ultrasonic sensing systems sense the motion of objects within the 
passenger seat area and determine whether the object is moving closer or 
5 farther away from the sensor location. However, ultrasonic sensors don't 
identify the nature of the object, and thus can't tell if it is a person or some other 
object, such as a bag of groceries. Similar to ultrasonic sensors, microwave 
sensors employed in active doppler radar systems can track objects, but cannot 
identify human presence. Weight sensors identify objects in the passenger 

10 seat based on applied pressure, but don't consider the passenger's location 
relative to the airbag. Passive IR sensors acquire thermal images of the 
passenger seat, but these systems are very costly. Present active IR sensors 
sense the relative location of the passenger relative to the airbag, but cannot 
identify human presence. Electromagnetic systems include LC resonant 

15 circuits where body capacitance is used to detect presence and identify objects, 
but these systems can not track the objects. 

U.S. Patent No. 5,835,613 issued to Breed et ai., November 10, 1998, 
discloses a vehicle interior monitoring system that claims to identify, locate and 
monitor persons in the passenger compartment of the vehicle. The monitoring 

20 system employs infrared emitters that illuminate the interior of the vehicle, and 
charge couple device (CCD) arrays that detect the radiation. Outputs from the 
CCD arrays are analyzed by computational devices that employ pattern 
recognition algorithms to classify, identify or locate the content or objects in the 
passenger seat. The pattern recognition system for determining vehicle 

25 occupants disclosed in the '61 3 patent employs complicated software that must 
learn the shape of an individual in alh kinds of lighting situations under various 
conditions. Additionally, employing pattern recognition in this manner is limited 
in its ability to track the individual as he or she moves around in the passenger 
seat Further, the ability to identify and track humans by general pattern 

30 recognition is questionably unreliable. Pattern recognition cannot identify who 
the person is, only detect an object's shape. 

What is needed is an improved airbag deployment sensor and system 
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that is able to effectively identify and track a person in the passenger seat of a 
vehicle. It is therefore an object of the present invention to provide such a 
sensing system for vehicle passenger airbag deployment. 

SUMMARY OF THE INVENTION 
5 In accordance with the teachings of the present invention, an imaging 

system for providing vehicle safety features is disclosed that employs face 
recognition software and algorithms to identify and track a person. The system 
includes infrared emitters that emit an infrared signal along a predetermined 
field-of-view, and an infrared sensor, such as a CMOS sensor used as a video 
1 0 signal array, that receives reflected infrared illumination from objects in the field- 
of-view. A processor, including the face recognition software, is employed to 
detect human faces. Once a face is detected, it can be compared to a data 
base to identify the person. 

Various applications of the imaging system for providing vehicle safety 
15 features include identifying the driver or passenger for personalizing the 
vehicle's airbags, providing pre-crash collision avoidance, providing blind spot 
detection, providing vehicle crash recording, and providing a warning signal if 
the driver appears drowsy. Additional objects, advantages and features of the 
present invention will become apparent from the following description and 
20 appended claims, taken in conjunction with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a side, cut-away, plan view of a person in the passenger seat 
of a vehicle in connection with an image sensing system for airbag deployment, 
according to an embodiment of the present invention; 
25 Figure 2 is a perspective view of a video camera and LED illuminator 

unit employed in the airbag deployment system of the present invention; 

Figure 3 is a graph with wavelength on the horizontal axis and luminous 
energy on the vertical axis showing the luminance curve for sunlight; 

Figure 4 is a representation of camera orientation and position 
30 parameters with respect to the center and normal of the airbag door; 

Figure 5 is a block diagram of the airbag deployment system of the 
present invention; 
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Figure 6 is a schematic diagram depicting a frame differencing technique 
used in the airbag deployment system of the present invention; 

Figure 7 is a timing diagram for the frame differencing technique of the 
invention; 

5 Figure 8 is a top plan view of a vehicle showing various sensor locations 

for sensing different areas of the vehicle for various security and convenience 
purposes, according to an embodiment of the present invention; 

Figure 9 is a side view of a vehicle showing a camera having a field-of- 
view for detecting objects in front of the vehicle, according to another 
1 0 embodiment of the present invention; 

Figure 10 is a top view of a vehicle including a side camera and a rear 
center camera for detecting objects at the side and rear of the vehicle, 
according to an embodiment of the present invention; 

Figure 11 is a side view of a vehicle including a camera positioned in 
15 front of the driver of the vehicle, according to another embodiment of the 
present invention; and 

Figure 12 is a side view of a vehicle including a camera mounted in the 
dome light of the vehicle for intrusion purposes, according to another 
embodiment of the present invention. 
20 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The following discussion of the preferred embodiments directed to an 
airbag deployment system that employs human facial feature recognition is 
merely exemplary in nature, and is in no way intended to limit the invention or 
its applications or uses. 
25 According to one embodiment of the present invention, a technique for 

detecting, identifying and tracking a persorr in the passenger seat of a vehicle is 
disclosed. As discussed above, it is sometimes desirable to deploy a 
passenger side airbag in a vehicle at low velocity, or not deploy the airbag, 
during a crash event, depending on whether a person is occupying the 
30 passenger seat of the vehicle, the proximity of the person to the airbag door, 
and the size of the person. According to the present invention, the identification 
and tracking of the person is determined by face recognition software, in 
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particular, software that recognizes and tracks the person's eyes and other 
facial features. Tracking can take place that way at many head angles and 
poses. The software algorithm would use calibrated face and feature 
separation to estimate range. The tracking of a localized area on the human 
5 face allows more image frames to be acquired per second, permitting more 
frequent tracking of the person. The more frequent tracking of the facial 
features will prevent acquiring a blurry image because the image is updated 
more frequently. 

Various software is known in the art that processes data from video data 
10 patterns received from an object being analyzed, and determines whether the 
object has a face. For example, such software includes the Visionics Face-It 
software, well known to those skilled in the art. It is stressed that the present 
invention is not limited to any particular facial feature mapping function, but can 
include any known algorithm, suitable for the purposes described herein, for 
1 5 recognizing facial features, whether it be two-dimensional or three-dimensional, 
that are then also to be used for ranging functions, as well. Further, according 
to the present invention, ranging algorithms are used in combination with the 
know face recognition software. 

As will be discussed in detail below, the present invention employs 
20 infrared radiation reflected off of objects in the passenger side of the passenger 
compartment of a vehicle that is received by an electronic video camera. The 
video camera generates the electrical signals and image used by the face 
recognition software to determine the presence identify and tracking of the 
person. 

25 Figure 1 is a cut-away, side, plan view of the passenger side 

compartment 10 of a vehicle 12. In this depiction, a person 16 is shown in the 
passenger seat 18, where the person 16 is moving forward during a crash 
event. An airbag pillow 20 is shown being deployed through an airbag door 24 
housed in an instrument panel 22 during the crash event. 

30 A video camera and IR LED illuminator unit 26 is mounted on the 

instrument panel 22 at a location suitable for the purposes described herein. 
Figure 2 is a perspective view of the unit 26 removed from the vehicle 12. The 
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unit 26 includes a cluster 28 of IR LEDs 30. A plurality of LEDs 30 is provided 
to generate the necessary intensity for daylight operation. The cluster 28 emits 
a beam of IR radiation towards the person 16 that is reflected therefrom back 
towards the unit 26. A video camera 34 is provided in the unit 26 to receive the 
5 reflected radiation from the person 16. The video camera 34 is used by way of 
a non-limiting example in that any detector that detects infrared radiation 
suitable for the purposes described herein can be used. 

A filter 36 is provided over the camera 34 to filter out radiation that is not 
within the desirable infrared range. The filter 36 can be any filter suitable for 

10 the purposes described herein, such as a Ti0 2 filter or a polarizing filter. The 
filter layers and thickness can be chosen to transmit the IR image to the 
detector, but to reflect the visible image away from the detector. The polarizing 
filter can be used to reduce visible light to the detector using electro-optical 
polarization that passes the IR wavelengths, but strongly attenuates the non-lR 

15 wavelengths. Figure 3 shows the luminance curve for sunlight, where the filter 
36 passes infrared radiation in the 40 nm bandpass window. The filter 36 
provides some protection against sunlight that may affect the operation of the 
airbag deployment system, and recognition of the face of the person 1 6. 

In this embodiment of the present invention, a single camera is used to 

20 acquire and monitor the range of the person 16. The software employed to 
perform this function utilizes two separate locations on the occupant's face to 
provide the ranging. In a preferred embodiment, it is the person's eyes that are 
detected to provide the triangulation for ranging purposes. However, as will be 
appreciated by those skilled in the art, other facial features of the person 16 can 

25 also be used, such as the person's ears, etc. Also, the software algorithm 
allows the person's head size to be determined so that both eyes do not need 
to be in view to track the person after he has been acquired. Additionally, the 
software can be used to view other parts of the person's body, such as the 
person's torso, in combination with the facial feature or head size detection. 

30 Because human facial features are unique, a database can be used to 

store specific information, such as eye-to-eye separation, about a person, so 
that the software can particularly identify that person. This is important, in one 
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example, so that the system can identify children and fifth-percent females, and 
inhibit air bag firing for these people, as required by government mandated 
regulations. Also, being able to specifically identify a person improves the 
ranging accuracy of the system because the system knows that person's eye 
5 separation or other specific facial features. 

The acquisition and tracking software needs to be calibrated for a 
particular seat position, airbag position, vehicle makeup, etc. Figure 4 is a 
representation of the camera orientation and position parameters with respect 
to the center and normal of the airbag door 24. The three parameters that must 

10 be estimated in the vehicle for ranging according to the invention after the 
camera and Visionics Face-It software are calibrated include two position 
offsets and one angular offset. The two position offsets are the lateral offset of 
the camera 34 with respect to the center of the airbag (AY), measured laterally 
and perpendicularly to the normal vector out of the center of the airbag door 24, 

1 5 and the front or back offset (aX) of the camera 34 with respect to the center of 
the airbag door 24, measured along an axis parallel to the normal out of the 
center of the airbag door 24. The angular parameter is the azimuth offset (A0) 
between the optical axis of the camera 34 and the normal vector coming out of 
the center of the airbag 20. Only two of these parameters, aX and A<t>, are 

20 used in the modified monocular distance equation. 

A calibration procedure, described below, can be used to determine AY, 
AX and AO- There is a trade off between measuring a large number of 
calibration points and linearly interpolating between them or measuring a small 
number of calibration points and non-Iinearly interpolating between them. In 

25 theory, calibrating with a large number of points would not require a model for 
estimating that is described herein. The cost of making the large number of 
measurements necessary for a brute force calibration and the possibility that 
out of range operation of the camera 34 or software might occur unnoticed with 
brute force calibration must be considered before using that approach. The 

30 approach used herein uses a small number of calibration tests in order to form 
a model for interpolation. 

From Figure 4, the following equations can be written in order to relate 
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measurements made in the airbag door reference frame (xyz) to those made in 
the camera lens reference frame (xyz). Only face positions in front of the 
airbag door (x>o) are considered in the following equations. For p ,0 (pitch or 
elevation angle) and <j> (yaw or azimuth angle) defined in the camera's spherical 
5 coordinate system, the equations for displacements x and y in the airbag 
coordinate system are: 

x=-AX+psin(e+A0)cos(<j)+A(j>) (1) 
y=-AY+psin(e+A0)sin(<),+A<|>) (2) 
This assumes that e + AG is fixed during the calibration. Rearranging equations 
10 (1) and (2) yields: 

Zlfl-tanC^A*) (3) 

Making eye coordinate readout measurements at fixed x+aX, and then taking 
the slope of the tangent with respect to changes in y yields: 



1 



jc + AX 



fixed _x = dtaa ^ A ^ (4) 



15 Knowing x and the result on the right side of equation (4), aX can be 
determined. Knowing AX, equation (3) and the measured data can be used to 
determine AY. Then, using AX and AY, equation (3) and the data, a§ can be 
determined. Using equation (3), and the fact that: 

eye_separation_distance=y rt _ eye -y teft _ eye , (5) 

20 the modified monocular equation follows from equation (6) for defining the 
objective parameter x, or the eye to airbag distance. 



x = -AX + 



eye separation _ dis tan ce 



(6) 

} _^ + A<D)-tanG> rt _^ + A<D) 

The calibration table consists of a lookup fable of the SDK determined 
eye coordinate as a pixel value linked to the associated ray slope, as viewed in 
25 the camera coordinate system. Equation (6) can be simplified in terms of ray 
slopes (tangents of singular angles), so that the calibration table can be used 
directly to determine the eye to airbag distance from monocular operation. 
Employing the tangent trigonometry identity in equation (7) below, 
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l-tan(a)tan(Z>) V ' 

and applying this identify to equation (6), leaves the modified monocular 
equation in a form that can be used by direct look up of tangent/slopes from the 
calibration lookup table. This equation is given in equation (8) below. 

eye _separation _ dis tan ce 



x = -AX + 



tanC^^+tanCAO) tan(^ rt ^ + tan(A<£) 



(8) 



l-tan(^ _^)tan(A<D) 1 - tan(# rt ^)tan(AO) 

Figure 5 is a block diagram of an imaging system 50 of the invention 
including a digital signal processor (DSP) 52. The DSP 52 includes the face 
recognition software and ranging functions that performs the analysis on the 
image generated by the camera 34. A clock generator 60 provides timing for 

10 the various digital devices in the system 50, and a power management system 
62 provides the power. The DSP 52 is connected to a CMOS chip 70 including 
a pixel array 72 that represents the IR detector, such as the camera 34. In this 
example, the pixel array 72 includes 256x256 pixels to provide the desired level 
of resolution. The CMOS chip 70 also includes various elements for timing and 

15 control purposes, including state matching logic 74, clock/timing circuitry 76, 
analog conditioning circuit 78, registers/buffers 80, on-chip programmable logic 
circuitry 82, etc. Additionally, an analog-to-digital converter 84 is also provided 
to convert the analog signal from the pixel array 72 to a representative digital 
signal. An on-chip SRAM memory 86 is shown for storage purposes, but can 

20 be off-chip as well. The operation of these devices in the system described 
herein would be apparent to those skilled in the art. 

In one embodiment, the infrared LEDs 30 are continuously on to provide 
the reflected radiation received by the camera 34. However, typically some 
kind of filtering or signal processing must be done to correct for the problems 

25 caused by direct sunlight on the camera 34 that make it through the filter 36. 
Particularly, the system 50 needs to be able to distinguish between shadows 
caused by sunlight and actual edges of features on the occupant 16. 
According to the present invention, a frame differencing technique is employed 
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that synchronously pulses the LEDs 30 on for a predetermined period of time 
and a predetermined number of frames of video data, and then off for a 
predetermined period of time over the same number of frames of video data. 
Then, the frames of data are subtracted from each other so that the frames 
5 without IR illumination can be subtracted from the frames with IR illumination, 
and the background can be eliminated. The detector is electronically shuttered 
synchronously with the pulses to provide exposure control. The frame 
differencing technique described herein is used in conjunction with the infrared 
pulsing to achieve the desired result. In other words, the frame differencing is 

1 0 synchronized to the infrared pulses. 

The concept of frame differencing is the time-aperture, pixel-level 
storage of images using natural illumination only and natural illumination with 
added infrared illumination. Frame differencing allows these images to be 
subtracted to mitigate the effects of strong visible illumination. The set-up 

15 includes a neutral density filter that sets the IR illumination, plus worst case 
background to maximize analog-to-digital converter input. Face recognition 
requires that the worst case analog-to-digital range for the differenced image be 
5 to 6 bits. The visible light would fit within the remaining range allowed by the 
analog-to-digital converter. The image differencing is either performed in the 

20 analog domain, where two pixel level capacitors are charged, one at each 
illumination level, or in the digital domain where a RAM memory of the digitized 
pixel output is taken at each illumination. The frame differencing acts to 
subtract out the illumination effects of the visible illumination, and to improve 
the image contrast The frame differencing function can be performed in the 

25 acquisition/high bandwidth generation mode or in the narrow bandwidth track 
mode using pulsed LED illumination. The number of the electrons from the 
pulsed IR light source must be 10 times greater than the photon noise of the 
ambient illumination. Here, the noise of the ambient illumination is the square 
root of two times the number of electrons within the sun intensity, because two 

30 image frames are being acquired for every one IR image received. 

Figure 6 is a representation of how the frame differencing is performed in 
the camera 34, according to one embodiment of the present invention. Figure 

10 
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7 is a signal timing line showing the operation of the frame differencing 
technique of the invention. A pixel array 90 in the camera 34 receives radiation 
from the scene for a predetermined time period (10 p,s) during a pulse of IR 
from the LEDs 30. At this time, the pixel array 90 receives ambient light and 
5 infrared light. The charge stored by each pixel or photodiode 92 in the array 90 
is then transferred to a charge storage site 94 made up of a plurality of 
capacitors, one capacitor 96 for each pixel 92. About 10 ^s later, at the end of 
the pulse from the cluster 28, the pixel array 90 detects just the ambient light for 
the same time period. The charge received by the pixel array 90 at this time 
10 period is stored in a capacitor storage site 98, having capacitors 100. An 
electronic shutter is employed in the detector to open and close at the 
appropriate time synchronously with the pulses of IR radiation for the operation 
described herein. 

The two storage sites 94 and 98 are summed in a summation amplifier 
15 102. The difference between the two storage sites 94 and 98 is then digitized 
by an analog-to-digital converter 104, and represents the frame of data where 
the ambient light has been removed. The readout of data takes about 1 0 ms, 
then at the next time period, the next pulse from the cluster 28 occurs. The 
complete frame differencing process can be performed on a single chip in 
20 CMOS where the pixel array 90 and the storage sites 94 and 98 are together. 
In an alternate embodiment, the frame differencing is performed at a different 
time period at an off-chip site, where the storage sites 94 and 98 are RAM. 

The frame differencing technique of the invention can be described in 
the following manner. The variables are defined as ambient illumination l(x,y), 
25 direct ambient illumination T(x,y), scene reflectance R(x, y), and modulated 
source as L when on, and 0 when off. The response of the camera 34 is 
proportional to the product of reflectance and illumination. 

S(x,y,OFF) = k*(l(x,y)*R(x,y)) + T(x,y) 
S*(x,y,ON) = k*((L+l(x,y)rR(x,y))+T(x,y) 
30 D(x,y) = S(x,y,ON)-S(x,y,OFF) = KL*R(x,y) 

This difference scene has much smaller dynamic range than the simple image 
S(x,y,OFF). The same benefit can be derived by reading out the frame with 
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LED, then reading the frame without LED and subtracting the frames external 
to the camera 34. The penalty is increased dynamic range required to avoid 
saturation. 

L must be much greater than the direct ambient illumination response on 
5 I. Thus, I is made as small as possible by using a narrow bandpass filter 
aligned in frequency to L. The raw sampling rate has to be twice the 
requirement set by the object tracking because two frames are differenced to 
get one frame to supply the face recognition software. The LEDs 30 have to be 
much faster. The IR radiation source has to be modulated such that all 

10 emission is during the time when all detectors are active. If the integration 
times of all pixels in the detector are not aligned, the available time for the 
source to be ON is reduced by the worst case misalignment. 

The face recognition system described above for identifying and tracking 
a person in a vehicle passenger seat can be extended to other types of vehicle 

15 systems for other purposes. Figure 8 is a top plan view of a vehicle 110 
showing sensors or cameras positioned at various locations for other purposes. 
For example, a forward looking infrared sensor 112 is positioned at a center 
location on the instrument panel 22 and is directed forward for detecting objects 
in front of the vehicle. Additionally, a rearward looking infrared sensor 1 14 is 

20 positioned at a center location on the instrument panel 22 and is directed 
towards the rear passenger compartment of the vehicle 110 to monitor activity 
in that part of the vehicle. Additionally, an infrared sensor 1 1 6 is positioned on 
the instrument panel 22 in front of the driver seat 118 to detect the driver's 
facial features for various purposes discussed below. An infrared sensor 120 is 

25 positioned on the instrument panel 22 in front of the passenger seat 122 for 
airbag deployment as discussed above. The various sensors 112, 114, 116 
and 120 can be used in different ways in different systems. Other sensor 
locations can also be employed for other 

purposes consistent with the discussion herein. Some of the applications are 
30 discussed below. 

In one embodiment, according to the present invention, a sensor or 
camera is pointed towards the passenger and driver's seat that takes an image 
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of the driver and the passenger's facial feature. Using the facial feature 
software, the facial features such as eyes, or any combination of 2-D or 3-D 
facial/head/torso feature coordinates of the driver and the passenger are 
compared to images stored in a computer data base or lookup table. If the 
5 software makes a match of the human face with one stored in the data base, 
the driver and/or passenger airbag will be adjusted so that that person's 
characteristics are used to personalize airbag deployment. This will help the 
airbag to automatically adjust for low weight, young persons, short drivers, etc. 
According to the invention, a drowsy driver application involves 

1 0 identifying when the driver is falling asleep at the wheel and waking the driver 
up. Initially, at regular time intervals, the face recognition software acquires the 
facial features and their coordinates of the driver. Depending on the new 
coordinates of the driver's facial features, the software will send a warning to 
the driver that he or she is falling asleep. This warning will be based on the 

15 frequency of the driver's head nods, i.e., the back and forth head motion, 
whether the driver's eyes are closed, etc. In particular, the percentage of time 
that the eyes are closed during a fixed time interval is used to determine if the 
driver is asleep. Preliminary studies have shown that when a driver's eyes are 
shut more than 70% of the time over two seconds, the driver is drowsy and may 

20 be falling asleep. In addition, the back and forth motion of the driver's head or 
the driver's eyes looking at a cell phone will be utilized to determine if the driver 
is paying attention to the road ahead. The face recognition software can sense 
the difference in the driver's eye coordinates and this information can be 
relayed back to the driver. This information returns to the driver in the form of a 

25 mechanical motion, or an electrical stimulus or the production of a loud signal to 
wake the driver. 

Similar to the drowsy driver application discussed above, drug use could 
cause the driver to fall asleep. Presently, many drivers use caffeine or speed to 
keep them awake over a long trip. However, when the drug wears off, the 
30 driver's response time diminishes to critical changes in road or driving 
conditions. Thus, monitoring of the driver's pupils will help detect a driver using 
drugs when the drugs stop working. Basically, the facial feature recognition 
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software determines how often the iris of the eye is blocked by the driver's 
eyelid to determine whether the driver is falling asleep. 

The facial feature recognition system described herein can also be used 
for driver identification. When the driver enters the vehicle, a camera pointing 
5 outward placed near the driver's door will take an image of the driver's facial 
features. Using the facial feature recognition software, the facial features such 
as eyes, or any combination of 2-D or 3-D facial/head/torso feature coordinates 
of the perspective driver are compared to images stored in the software data 
base. If the software makes a match of the face with one stored in the data 

10 base, the doors will open and the vehicle can be automatically started. 
Otherwise, the vehicle will remain in the off position. In an alternate 
embodiment, after the driver gains access to the vehicle and is in the driver 
seat, the camera can be used to identify the driver for purposes of starting the 
vehicle. Additionally, in a parental control situation, the driver can be identified 

15 as a young person, where the algorithm sets the control of the vehicle to a 
maximum speed for that driver's use. 

The driver ID concept can be extended to driver convenience features 
using the face recognition software as well. Particularly, the driver's human 
facial features can be used to recall the driver's last seat position, rear and side 

20 mirror adjustment, steering wheel setting, radio setting, etc. In this example, 
the vehicle's computer can store the driver's last seat position, and rear and 
side mirror adjustment with the driver's human facial features. Subsequently, 
whenever the camera acquires the driver's facial features and this image 
matches a stored driver's personal preference, the seat and mirrors can be 

25 automatically repositioned to the desired locations. 

The use of the specific imaging and processing techniques of the 
present invention allows the presence detection of people of all ages and sizes, 
whether conscious or asleep, in the interior of a vehicle with very high reliability. 
This detection is effective at all times, for stationary or moving vehicles, 

30 whether the engine is running or not, without being perturbed by external 
stimulus to the vehicle activities so that determination of dangerous human 
interior situations is enabled to generate warning alerts locally or to be 
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communicated. This includes situations of abandoned children, medical 
emergency, captive occupants, sleeping occupants, etc. 

Figure 9 is a side view of a vehicle 130 including a forwarded mounted 
sensor 132 that provides a sensor field-of-view in front of the vehicle 130. 
5 Reflections from objects in the forward looking direction, such as a pedestrian 
in a crosswalk, a slow moving vehicle, etc., will be registered by the processing 
system, and provide a warning to the driver of an object in the vehicle's path. 
By recognizing a pedestrian in a crosswalk or the like, pre-crash avoidance can 
be provided using the face recognition system of the present invention. 

10 Figure 10 is a top view of a vehicle 140 that includes a center mounted 

camera 142 behind the vehicle 140 for providing an indication of objects in the 
vehicle's path when it is moving in a rearward direction. Additionally, a camera 
144 placed in the driver's side mirror 146 can detect objects located in the 
driver's blind spot to provide warnings that the driver may not see. In this 

15 example, the camera 144 will allow the driver to stay focused on the road 
ahead, but can still safely make a right or left turn or lane change. The camera 
144 can be provided for detecting pedestrians or riders on bicycles in the 
vehicle blind spot by the face recognition system. 

In severe automobile collisions, it is sometimes difficult to determine 

20 what caused the accident. In one example, before the crash event, the exterior 
forward looking camera can make pre-crash images. These images will be 
relayed back to the driver in the form of an electrical or mechanical warning so 
that the driver can make adjustments to avoid a collision. In the event of a 
vehicle collision, the images acquired will be stored so at a later date, the pre- 

25 crash sensing conditions will be available to make a post crash injury 
assessment. After the crash, a crash notification signal can be automatically 
transmitted in the form of a radio wave signal which will alert the police and fire 
department of the crash location. 

To keep the driver's attention on the road and hands on the steering 

30 wheel, the driver's facial gestures, such as movement of facial features, can be 
used to perform tasks previously performed manually. For example, a driver's 
facial feature command can be compared to an existing stored facial feature 
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that will cause the vehicle to automatically change the automobile controls 
without the driver taking his or her eyes off the road. Some examples include 
control of automobile accessories, such as turning on and off the radio, turning 
on and off the vehicle lights, turning on and off the windshield wipers or blowing 
5 the horn. Figure 1 1 shows a side view of a vehicle 152 including a sensor 154 
of the type discussed herein positioned in front of a driver 156 for identifying 
and tracking the facial features of the driver 156. The face recognition system 
of the invention can use this information in the manner as discussed herein. 

Further, a camera can be positioned in the vehicle to monitor intrusion. 

10 In this example, a camera takes an image anywhere inside of the vehicle 
passenger compartment. If a person is detected in the backseat of the vehicle, 
the camera will take an image of the person's facial features. If the facial 
features match one stored in the computer, then no alarm will go off. 
Otherwise, an alarm can be sounded to alert the driver that someone is in the 

15 back seat. The camera can provide a recording of the intruder for later 
identification, and provide detection of human presence after the vehicle is 
locked. Figure 12 shows a side view of a vehicle 160 depicting this example, 
where a sensor 162 is mounted at a center location in the vehicle passenger 
compartment, such as in the center dome light. 

20 The foregoing discussion discloses and describes merely exemplary 

embodiments of the present invention. One skilled in the art will readily 
recognize from such discussion and from the accompanying drawings and 
claims, that various changes, modifications and variations can be made therein 
without departing from the spirit and scope of the invention as defined in the 

25 following claims. 
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CLAIMS 

What is Claimed is: 

1 . An imaging system for a vehicle, said system comprising: 

an infrared source generating an infrared beam along a 
particular field-of-view in or around the vehicle; 
5 an infrared detector responsive to infrared radiation reflected 

from objects in the field-of-view of the infrared beam, said detector generating 
image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
said processor including face recognition software that images facial features 
10 of a person to detect a person's face, said processor comparing the detected 
image of the person's face with stored images, and causing a vehicle 
operation to be performed if a particular person's face is identified. 

2. The system according to claim 1 wherein the processor 
personalizes an airbag for a particular person that is detected. 

15 3. The system according to claim 1 wherein the processor 

monitors the eyes of the person, and provides a signal if the person appears 
drowsy. 

4. The system according to claim 3 wherein the processor 
monitors the time the person's eyes are closed. 
20 5. The system according to claim 3 wherein the processor 

monitors the person's head nods. 

6. The system according to claim 1 wherein the infrared detector is 
a cluster of infrared light emitting diodes. 

7. The system according to claim 6 wherein the infrared detector is 
25 an electronic video camera. % 

8. The system according to claim 7 wherein the cluster of LEDs 
and the electronic camera are provided as a single unit mounted within the 
vehicle. 

9. The system according to claim 1 wherein the processor tracks 
30 and monitors the persons eyes. 

10. The system according to claim 1 further comprising an infrared 
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filter positioned in front of the detector. 

11. The system according to claim 1 wherein the infrared detector 
includes a pixel array of photodiodes. 

12. The system according to claim 1 wherein the infrared source is 
5 pulsed on for a predetermined time at predetermined intervals. 

13. The system according to claim 12 wherein the pulsed source 
and the detector are synchronized such that a difference of successive 
frames represents an image free of ambient light variations. 

14. The system according to claim 1 wherein the detector includes a 
1 0 pixel array of diodes. 

15. An airbag personalization system for a vehicle, said system 
comprising: 

an infrared source generating an infrared beam along a 
particular field-of-view in or around the vehicle, said source being pulsed on 
1 5 and off at predetermined intervals; 

an infrared detector responsive to infrared radiation reflected 
from objects from the field-of-view of the infrared beam, said detector 
generating image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
20 said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor comparing the detected 
image of the person's face with stored images, said processor personalizing a 
vehicle airbag for a particular person that is identified. 

16. A drowsy driver detection system for a vehicle, said system 
25 comprising: 

an infrared source generating an infrared beam along a 
particular field-of-view in or around the vehicle, said source being pulsed on 
and off at predetermined intervals; 

an infrared detector responsive to infrared radiation reflected 
30 from objects from the field-of-view of the infrared beam, said detector 
generating image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
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said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor comparing the detected 
image of the person's face with stored images, said processor monitoring the 
eyes of the person and providing a signal if the person appears drowsy. 
5 17. The system according to claim 16 wherein the processor 

monitors the time the person's eyes are closed. 

18. The system according to claim 16 wherein the processor 
monitors the person's head nods. 

19. An imaging system for a vehicle, said system comprising: 

10 an infrared source generating an infrared beam, said infrared 

source being mounted at a forward location of the vehicle and providing a 
field-of-view in front of the vehicle; 

an infrared detector responsive to infrared radiation reflected 
from objects in the field-of-view of the infrared beam, said detector generating 
15 image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor providing a signal if a 
person's face is detected in front of the vehicle. 
20 20. An imaging system for a vehicle, said system comprising: 

an infrared source generating an infrared beam along a 
particular field-of-view in or around the vehicle; 

an infrared detector responsive to infrared radiation reflected 
from objects in the field-of-view of the infrared beam, said detector generating 
25 image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor providing a signal if a 
person's face is detected in a blind spot of the vehicle. 
30 21 . An imaging system for a vehicle, said system comprising: 

an infrared source generating an infrared beam along a 
particular field-of-view in or around the vehicle; 
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an infrared detector responsive to infrared radiation reflected 
from objects in the field-of-view of the infrared beam, said detector generating 
image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
5 said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor providing a recording of 
people within or around the vehicle during a crash event 

22. An imaging system for a vehicle, said system comprising: 

an infrared source generating an infrared beam along a 
10 particular field-of-view in or around the vehicle; 

an infrared detector responsive to infrared radiation reflected 
from objects in the field-of-view of the infrared beam, said detector generating 
image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
15 said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor providing a signal if an 
incapacitated person is detected in the vehicle. 

23. An imaging system for a vehicle, said system comprising: 

an infrared source generating an infrared beam along a 
20 particular field-of-view in or around the vehicle; 

an infrared detector responsive to infrared radiation reflected 
from objects in the field-of-view of the infrared beam, said detector generating 
image signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
25 said processor including face recognition software that images facial features 
of a person to detect a person's face, said processor monitoring gestures of 
the detected person to cause a vehicle operation to be performed. 
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